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VIDEO DISPLAY SYSTEM WITH TWO CONTROLLERS EACH ABLE TO 
SCALE AND BLEND RGB AND YUV SURFACES 

This application is related to a commonly assigned co-pending application filed 
5 simultaneously herewith (docket number 10442-6US) entitled "MULTIPLE 
DISPLAY DEVJCE DISPLAY CONTROLLER WITH VIDEO OVERLAY AND 
FULL SCREEN VIDEO OUTPUTS", the specification of which is hereby 
incorporated by reference. 

10 Field of the Invention 

The present invention relates to a single controller system for multiple display 
output. 

Background of the Invention 

15 Video display controllers are known which are able to work with both RGB format 
video and YUV format video to combine two video surfaces from graphics 
memory to provide an overlay output. This is useful, for example, in the situation 
that a movie source in YUV format is obtained by decoding a video file from say 
a CD-ROM or DVD device, and written to a video surface in graphics memory. At 

20 the same time a computer environment desktop is stored in RGB format in 
another surface within graphics memory. These two surfaces can be combined to 
provide the YUV source within a window on the desktop. Because the video 
display controller handles the conversion, scaling and mixing of the two sources, 
the computer operating system or application program does not directly create 

25 the desired display, but rather instructs the display controller system to scale and 
combine the two sources as desired. The quality of display and efficiency of the 
computer system is thereby greatly enhanced. An example of such a system is 
described in US patent 5,764,201 to Ranganathan. 

30 In such displays controller systems, only one combined output of mixed format 
video is possible in which case the second output can only mirror what is being 
displayed by the first output Furthermore, since only two surfaces (one RGB and 
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one YUV) are being fetched one for each of only one RGB pixel path and only 
one YUV pixel path, the degree of flexibility is further limited, and it would be 
desirable to have greater control over surface selection, combination, scaling and 
output. 

5 

Summary of the Invention 

It is therefore a first object of the present invention to provide an apparatus and a 

method for controlling a single graphic controller system with a single memory 
cni££2£liJ "* v ,!!iS *l i w< ? c ! ,s P , . a y outputs, in which control of surface 
10 selection, conversion, combination, scaling and output is afforded without the 
limitations known in the prior art. 

According to the invention, there is provided a method of providing a display 
output for at least two display devices using a single controller system. The 

15 method comprises: 

providing a first display controller able to read from a graphics memory at 
least two first surfaces into at least two first pixel paths, convert the pixel format 
of at least one of the at least two first surfaces, scale at least one of the at least 
two first surfaces, and combine (blend and/or overlay) the at least two first 

20 surfaces, where at least one of the first pixel paths supports any one of RGB and 
YUV pixel format and at least one of the pixel paths supports at least RGB pixel 
format; providing a second display controller able to read from a graphics 
memory at least two second surfaces, convert the pixel format of at least one of 
the at least two second surfaces, scale at least one of the at least two second 

25 surfaces, and combine the at least two second surfaces, where at least one of 
the pixel paths supports any one of RGB and YUV pixel formats and at least one 
Of the pixel paths supports at least an RGB pixel format- 
causing the first display controller to select and read the first surfaces, 
convert the first surfaces into a like first format at least when the first surfaces are 

30 not all in the like first format, scaling at least one of the first surfaces, combining 
(blending and/or overlaying) the first surfaces to obtain a blended first surface, 



2 



MAP, 16. 2000 9: 19PM 



SWaBEY OG i LV i MIL 514 m Soby 



n u. bo f y r. n.'ii 



10442-5US 

and outputting the combined first surface to provide a first output stream of pixel 
data; 

causing the second display controller to select and read the second 
surfaces, convert the second surfaces into a like second format at least when the 
5 second surfaces are not in the like second format, scaling at least one of the 
second surfaces, combining (blending and/or overlaying) the second surfaces to 
obtain a blended second surface, and outputting the combined second surface to 
provide a second output stream of pixel data, in this way, flexibility is provided 
by selection of the first and second surfaces as well as scaling and combining of 
10 the first and second surfaces, whether the surfaces are in RGB format, YUV 
format or mixed RGB/YUV format. It will be appreciated that any of the surfaces 
could be common between the two controllers. This enables the same surface to 
be displayed on the two displays in different ways, formats and blended with 
different surfaces. 

15 

While the preferred embodiment suggests that any of the scalers be part of the 
controllers, the functioning of any of these scalers can be earned out by using the 
scaling capabilities of the graphic controller system's drawing engine whereby 
that particular scaler may be removed from the preferred embodiment. 

20 

Preferably, the first output stream and the second output stream are fed into a 
first multiplexer and a second multiplexer, an output of the first multiplexer being 
fed into a first display, and an output of the second multiplexer being fed into a 
second display, and the method further comprises causing the multiplexers to 
25 select a desired one of the output streams for display on the first and second 
displays. 

Also preferably, the first display controller reads two first surfaces, has at least 
one controllable color space converters outputting a selected one of RGB and 
30 YUV pixel format , and two scaling units scaling an output of the two color space 
converters, and a combining unit receiving an output of the two scaling units. The 
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method then preferably includes causing the two scaling units to scale each of 
the two first surfaces. While the preferred embodiment includes two scalers, 
fewer or more scalers can be used. As an example, the two scaling units can be 
replaced by one scaler scaling the output of the combining unit in a case where it 
5 is known that the two surfaces do not need to be scaled independently. 

The single controller system may alternatively comprises a drawing engine scaler 
instead of any one or all backend scalers responsive to a scaling command to 
pre-scate at least one surface in the graphics memory and output a scaled 
version in a scaled surface in the graphics memory; and one of the steps of 
1 0 causing the first display controller and causing the second dispJay controller may 
comprise one of scaling the at least one of the first surfaces and the at least one 
of the second surfaces, respectively, using the drawing engine and reading the 
at least one of the first surfaces and the at least one of the second surfaces, 
respectively, from the scaled surface. 

15 

While the description of the invention uses two controllers as the preferred 
embodiment, it can easily be extended or scaled to additional controllers. 

Two independent display controllers (primary display and secondary display 
20 controller) each able to generate a stream of pixel data and associated 
synchronization signals (syncs) from pixel data contained in a display memory 
(surfaces). Each display controller can combine (overlay or blend) two or more 
surfaces using two or more pixel paths with or without scaling in a wide variety of 
pixel formats and color spaces. (Note that the number of surfaces can be more 
25 than two while still remaining within the scope and spirit of the invention). This 
provides a great deal of capability and flexibility, which enables a host of useful 
new features to the end user. The two display controllers can drive a vahefy of 
output ports including any combination of RGB D/A converters, video encoder, 
and TOMS Panel Link or LCD interface. This permits many display combinations 
30 such as two RGB monitors, two TV monitors, two flat panel displays or any mix 
of them. 
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The primary display controller can read two surfaces from display memory into 
two pixel paths. Each surface can be in any one of a variety of pixel data formats 
such as YUV (4:2:0, 4:2:2 etc.) and RGB (8. 16, 24, 32). The primary display 
5 controller can combine (overlay or blend) the two surfaces together. One or both 
of the surfaces can optionally be re-scaled using either one of or all of the 
backend scalers before or after being combined using the appropriate backend 
scaler or the 30 drawing engine into a display surface in memory before even 
being read by the display controller. A fixed or programmable color space 

10 converter is included in one or both pixel data paths, so that combined pixels are 
in the same color space. For example, a surface can be in YCbCr or YUV format 
(video) and is converted in RGB (graphic) format before being combined with the 
RGB Windows desktop. Typically, but not exclusively, this is used to display a 
video surface (video in a window) under a graphic overlay surface (e.g. 2D 

15 desktop). One of the overlay surface can be a graphic bitmap. Typically, but not 
exclusively, this is used to display a DVD sub-picture overlay (or blended) 
surface over a DVD decoded picture. 

The secondary display controller can also simultaneously read 2 surfaces from 
20 display memory into two pixel paths and combine them together. The secondary 
display controller can be identical to the primary display controller, or it can 
support a subset of the full feature set . For example, the secondary display 
controller's main surface could either be in YCbCr (4:4:4, 4:2:2, 4:1:1, 4:2:0) or 
RGBalpha (8, 16, 24, 32 bpp) format which allows for various types of surfaces to 
25 be displayed (grabbed video from video input port, S/W decoded video, graphic 
images etc.). The second overlay surface can be a graphic bitmap or other type 
of surface. Typically, but not exclusively, this is used to display a DVD sub- 
picture overlay surface over a DVD decoded picture. Note that in single chip 
multi-display controllers of prior art, blending and scaling units are limited to one 
30 controller only and/or a total of only two pixel paths are present . This invention 
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requires the incorporation of at least one of such units and at least two pixel 
paths in each of at least two controllers in order to increase flexibility. 

One or more CRTCs can be set to convert a 16 or 32 bpp RGBaipha format into 
a YUV 4:2:2 + alpha format. When RGB32 is output as YUV 4:2:2 + alpha data 
(upper 8 bits out of RGB32), the alpha data is useful as a key signal. This mode 
is useful for outputting a Windows desktop surface as a standard video signal. It 
is also useful for using the video controller as a source of graphics plus key for 
professional video graphics and titling applications. 

One of the main applications of the invention is where the second display can 
show a video image full screen while the main display maintains a separate 
image such as a fully interactive Windows desktop. The video from the second 
display can also be replicated on the first display within a window. For the video 
on both displays to have sub-picture blending, both controllers must have sub- 
picture blending units. 

Brief Description of the Drawings 

The invention wilf be better understood by way of the following detailed 
20 description of a preferred embodiment with reference to the appended drawings, 
in which: 

Fig. 1 is a schematic block diagram of a single controller system for 
providing dual display output according to the preferred embodiment; and 

Fig. 2 is a schematic block diagram of a controller reading two surfaces of 
25 any of a plurality of video formats, carrying out conversion, if necessary, and 
scaling and combining (overlaying and/or blending) in accordance with the 
preferred embodiment; and 

Fig. 3 is a pictoral diagram of two different surfaces in graphics memory 
showing the result of combining with sub-picture blending and overlaying. 

30 

Detailed Descrip tion of the Preferred Embodiment 
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Figure 1 shows a high level block diagram of the invention. Two CRTC 
controllers (11) and (12) are capable of fetching 2 surfaces from memory (50) 
which can be SGRAM, SDRAM, or any other type of Random Access Memory. 
These surface can be in a variety of pixel formats including but not limited to 
5 RGB (8, 16, 24, 32) and YUV (4:2:0 , 4, 2, 2). The output of each CRTC can be 
displayed on a CRT monitor, TV or flat Panel displays via appropriate converters, 
encoders and transmitters. The Multiplexers (33) and (34) allow routing of the 
output of each CRTC to either display. This allows either display to receive the 
output from either CRTC. 

10 

Each CRTC contains one or more backend Scaler (14) (refer to Figure 2) that 
allows the input surfaces to be re-scaled before being combined (overlaid, 
blended etc.) with the other surface. Alternatively, prescaling of any surface could 
be done by the drawing engine . A fixed or programmable color space converter 

15 (15) is included in one or both pixel data paths, so mat combined pixels are in the 
same color space. For example, a surface can be in YCbCr format (video) and is 
converted in RGB (graphic) format before being combined with the RGB 
Windows desktop. Typically, but not exclusively, this is used to display a video 
surface (video in a window) under a graphic overlay surface (e.g. 2D desktop). 

20 Each CRTC also contains a combining unit (or sub picture blending unit) for 
combining (overlaying or blending) the two surface. 

Typically, the graphics memory 50 and one or all encoders are external of the 
discrete device (ASIC) containing the CRTCs, the drawing engine and the 
25 graphics subsystem manager. 

Although conversion to RGB format is illustrated and preferred over conversion 
to YUV format, both types of conversion are possible. 

30 Figure 3 shows the results of combining a video surface with another surface (in 
this case a bitmap) using sub-picture blending (top right) or sub picture overlay 
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(bottom right). One application of this would be adding logos, subtitles etc. Note 
in the case of DVDs (as an example) where the language of the subtitles is user 
selectable, the sub-titles are not embedded in the source of the YUV surface but 
are overiaid in realtime from a user selectable source. 

The operation and design of the blending units and other units within the CRTCs 
(Scaler 14 and CSC 15) are general and well known to someone skilled in the art 
and are therefore not described here. 

Instead of using the backend scaler, the 3D drawing engine 60 of the graphics 
controller can also be used for pre-scaling surfaces and storing them in memory. 
While the scaling capabilities of the 3D drawing engine are typically used for 
scaling textures for 3D objects, the capability can be leveraged to scale any type 
of surface by testing the surface as a texture. This also allows a CRTC having 
only one scaler to blend surfaces both of which need to be scaled, and this 
allows a CRTC having no scaler to blend surfaces in which at least one of the 
surfaces has been scaled prior to blending. 

It will be appreciated that CRTC1 and CRTC2 each read two surfaces from one 
memory 50, and that one or both of these surfaces may be the same surfaces in 
which case the same surface can be displayed in different ways. The present 
invention is also not limited to reading only two surfaces into two pixel paths, but 
a CRTC may be designed to be controlled to read three or more surfaces into 
three or more pixel paths. 
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What is claimed is: 

1 . A method of providing a display output for at least two display devices 
using a single graphic controller system, the method comprising: 

providing a first display controller able to read from a graphics memory at 
least two first surfaces into at least two pixel paths , convert at least one of the at 
feast two first surfaces, scale at least one of the at least two first surfaces, and 
combine (blend and/or overlay) the at least two first surfaces, said first surfaces 
containing any one of RGB and YUV format video , 

providing a second display controller able to read from a graphics memory 
at least two second surfaces into at least two pixel paths, convert at least one of 
the at least two second surfaces, scale at least one of the at least two second 
surfaces, and combine (Wend and/or overlay) the at least two second surfaces, 
said second surfaces containing any one of RGB and YUV format video; 

causing said first display controller to select and read said first surfaces, 
convert said first surfaces into a like first format at least when said first surfaces 
are not all in said like first format, scaling at least one of said first surfaces, 
combining said first surfaces to obtain a combined first surface, and outputting 
said combined first surface to provide a first output stream of pixel data; 

causing said second display controller to select and read said second 
surfaces, convert said second surfaces into a like second format at least when 
said second surfaces are not in said like second format, scaling at least one of 
said second surfaces, combining said second surfaces to obtain a combined 
second surface, and outputting said combined second surface to provide a 
second output stream of pixel data, 

whereby flexibility is provided by selection of said first and second 
surfaces as well as scaling and blending of said first and second surfaces, 
whether said surfaces are in RGB format, YUV format or mixed RGB/YUV 
format. 



9 



16.2000 9:2 1PM 



SWABEY OGILVY MTL 514 238 8 J89 



llu. bo f y r l 



10442-5US 

2. The method as claimed in claim 1 . wherein the first output stream and the 
second outpui stream are fed Into a first multiplexer and a second multiplexer, an 
output of the first multiplexer being fed into a first display, and an output of the 
second multiplexer being fed into a second display, the method further 
comprising causing said multiplexers to select a desired one of said output 
streams for display on said first and second displays. 

3. The method as claimed in claim 1, wherein: 

said first display controller reads two first surfaces, has at least one 
controllable color space converters outputting a selected one of RGB and YUV 
format video, and one scaling units scaling an output of said said at least one 
color space converters and another scaling unit independently scaling the 
unconverted surface, and a combining unit receiving an output of said two 
scaling units, 

the method comprising causing said two scaling units to scale each of said 
two first surfaces. 

4 The method as claimed in claim 1 , wherein: 

said first display controller reads two first surfaces, has at least one 
controllable color space converters outputting a selected one of RGB and YUV 
format video, and one combining unit combining an output of said at least one 
color space converters and another surface, and a scaling unit receiving an 
output of said combining unit, 

the method comprising causing said scaling unit to scale said combined 
surfaces. 

5 The method as claimed in claim 1 , wherein: 

said second controller reads two second surfaces, has at least one 
controllable color space converters outputting a selected one of RGB and YUV 
format video, and one scaling units scaling an output of said at least one color 
space converters and another scaling unit independently scaling the un 



10 



M. 16.2000 9:22PM 



SWaBEY OGILvY MTL 514 IW 8 Jby 



moo /a r. wl\ 



10442-5US 

converted surface, and a combining unit receiving an output of said two scaling 
units, 

the method comprising causing said two scaling units to scale each of said 
two second surfaces. 

6 The method as claimed in claim 1 , wherein: 

said second display controller reads two second surfaces, has at least one 
controllable color space converters outputting a selected one of RGB and YUV 
format video, and one combining unit combining an output of said at least one 
color space converter and another surface, and a scaling unit receiving an output 
of said combining unit, 

the method comprising causing said scaling unit to scale said combined 
surfaces. 

7. The method as claimed in claim 1 , wherein; 

said single graphic controller system comprises a drawing engine scaler 
responsive to a scaling command to pre-scale at least one surface in said 
graphics memory and output a scaled version in a scaled surface in said 
graphics memory; and 

one of said steps of causing said first display controller and causing said 
second display controller comprises one of scaling said at least one of said first 
and second surfaces, respectively, using said drawing engine scaler and reading 
said at least one of said first and second surfaces, respectively, from said scaled 
surface wherein at least one of said first and second controllers does not have at 
least one of said backend scalers. 
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ABSTRACT 

Multiple independent displays from one graphics controller can be driven in a 
wide variety of modes using multiple display controllers. This invention 
incorporates a unique combination of enhancement building on a generic single 
multi-display graphics subsystem. Digital video is gaining vast popularity in the 
worldwide consumer, professional and commercial market. The applications of 
this invention cover ali these market segments and include (but are not limited to) 
Digital Content Creation, Digital video playback, digital entertainment, DVD 
Authoring etc etc. 
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I believe I am the original, first and sole inventor (if only one 
name is listed below) or an original, first and joint inventor (if 
plural names are listed below) of the subject matter which is 
claimed and for which a patent is sought on the invention 
entitled 
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SCALE AND BLEND RGB AND YUV SURFACES". 



et dont le memoire descriptif est ci-joint a moins que k case 
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de demande des £tats-Uni$ ou le num&o de demande 

Internationale PCT __, ^ et 

modifies le (le cas echeant). 

Je declare par la presente avoir revise* et compris ie contenu du 
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revendications, telles que modiflees par toute modification ci* 
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following box is checked: 

□ was filed on _^as United States 

Application Number or PCT International Application 

Number, m m t and was amended 

on -.. (if applicable). 

I hereby state that I have reviewed and understand the contents 
of the above identified specification, including the claims, as 
amended by any amendment referred to above. 



Je reconnais devoir divulgner toute information petrinente a la 
brevetabilite ? tel que defini dans le Titre 37^ §1.56 du Code 
federal des reglementations. 
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j l ai aussi indique ci-dessous toute demande etrangere de brevet, 
tout certificat tfinventeur oa toute demande interaationale PCT 



1 acknowledge the duty to disclose information which is 
material to patentability as defined in Tide 37, Code of Federal 
Regulations^ §1.56. 

I hereby claim foreign priority under Tide 35> United States 
Code, §I19(a)-(d) or §365 (b) of any foreign application^) for 
patent or inventor's certificate, or §365{a) of any PCT 
International application which designated at Jeast one country 
other than the Orated States, listed below, and have also 
identified below, by checking die box, any foreign application 
for patent or inventor's certificate, or PCT International 
application having a filing date before that of the application 
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(Application No./ N° de demande) 



(Filing Date/ Date de d&pdt) 



(Application No./ N° de demande) 



(Filing Date/ Date de depot) 
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35, §120 du Code des J&tats-Unis, de toute demands de brevet 
effectode avix £tat$-Uni$, ou en vertu du Titre 35 } §365(c) du 
meme Code, de toutc demandc international PCT designant 
les fetats4Jnts et figurant ci-des$ou$ et, dans la mesure ou 
Tobjet de chacune des revendications de ceite demande de 
brevet n'est pas divulgue dans la demande anterieure 
americaine ou Internationale PCT, en vertu des dispositions du 
premier paragraph© du Titre 35, §112 du Code des Etats-Unis, 
je reconnais devoir divulguer toute information pertinente a la 
brevetabiiite 3 tei que defini dans le Titre 37, §1-56 du Code 
federal des regimentations, dont j'ai pu disposer entre la date 
de d^pdt de la demande anterieure et la date de d£pdt de la 
demande nationals ou Internationale PCT de & pr&ente 
demandc: 



I hereby claim the benefit under Title 35, United States Codc ? 
§120 of any United Stales applications), or §365(c) of any 
PCT International application designating the United States, 
listed below and, insofar as the subject matter of each of the 
claims of this application is not disclosed in The prior United 
States or PCT International application in the manner provided 
by the first paragraph of Title 35, United Sates Code, §112, 1 
acknowledge the duty to disclose infonnation which is material 
to patentability as defined in Title 37, Code of Federal 
Regulation^ §1.56 which became available between the filing 
date of the prior application and the national or PCT 
International filing date of this application. 



(Application No.) 
(N* de demande) 



(Filing Date) 
(Datededepdt) 



(Status) (patented, pending, abandoned) 
(Statat) (brevet^, en cours d'examen, abandonn£) 



(Application No.) 
(W de demande) 



(Filing Date) 
pate de depot) 



(Status) (patented, pending abandoned) 
(Statut) (brevet^, en cours d'examen, abandon^) 
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French Language Declaration 

Je declare que toute les declarations faites dans la presents sont 
i ma conaaissance, v6ridiques et que toutes les declarations 
faites a partir de renseignements ou de suppositions sent tenues 
pour v£ridiques; et de plus, que touies ces declarations ont et£ 
faites en sachant que touts fausse declaration voloataire ou son 
equivalent est passible d'une amende ou d'une peine 
d'emprisonnement, ou des deux, en vertu de & Section 1001 du 
Titre 18 du Code des Etais-Unis, et que de telles declarations 
volontairement fausses risquent de compromettre la validity de 
la demande de brevet ou du brevet d&ivre a partir de celle-ci. 

POUVOIR: En tant qu'inventcur disigne, Je noxnme par la 
pxisente V(les) avocat(s) ctfou agent(s) inclus sous 1e numcto 
de client officiel suivant^ avec plein pouvoir de revocation et 
de substitution, charges de poursuivre cette demande et de 
traiter route affaire s'y rapportaat avec TOffice des brevets et 
des marques: (menUonner h nom et h nwniro 
d'enregistrement). 



Please send all correspondence and direct all Telephone calls to: / Ycuillez adresser toute correspondance ct tout appel telephonique k 

iinv 

020988 



Full name of soJe or first inventor (Nom complet de Vunique ou premier inventenr) 

Kamran AHMED 


Citizenship (Nationalise) 

Canadian 


E&tc (ddtemityyyy) (jj/mm/aaaa) 


Residence and Post Office address (Dottdcile et adiesst-p^tstW) 

418 Pine Avenue West, Apt, 1 s 
Montreal, Quebec, CANADA H2W 1S2 


Inventors signature (signature de linveatetir) 


Full name of second inventor {Nom complex du second ca-iaventeur) 


Citizenship (Nationality 


Date (dd/mm/yvyy) (jjAnm/aaaa) 


Residence and Post Office address (Domicile et adresse postale) 


Second fcwemor's signature (signature du second invemeur) 


Full name of third co-mventor (Nom complex du troisicme co*inventeur) 


Citizenship (Nationality) 


Date (dd/mm/yyyy) Gj/mm/aaea) 


Residence and Posi Office address (Domicile ct adressc postale) 


Third Inventor's signature (signatured*/ troisi&ne invcatear) 


Full name of fourth co-mvemoi (Nom compla du qpatrieme co*invenrear) 


Citizenship (Nationalist) 


Date (dd/wmfyyyy) (y%m/aaaa) 


Residence and Post Office address (Domicile et adressc posiale) 


Fourth Inventors signature (Signature duquatrteme invemeur) 



flu. bB'/U r. 



I hereby declare feat all statements made herein of my own 
knowledge are true and that all statements made on information 
and belief are believed to be true; and further that these 
statements were made with the knowledge ihat willful false 
statements and the like so made are punishable by fine or 
imprisonment, or both, under Section 1001 of Title 18 of the 
United States Code and that such willful false statements may 
jeopardize the validity of the application or any patent issued 
thereon. 



POWER OF ATTORNEY: As a named inventor, I hereby 
appoint the agents and/or attorneys included in the following 
Customer Number, with full power of substitution, 
association, and revocation, to prosecute this application and 
to transact all business in 4c Paxeni and Trademark Office 
connected therewith; (list name and registration number) 




page 3 of 3 



